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THE MAILING DATE OF THIS COMMUNICATION. 

- Extensions of time may be available under the provisions of 37 CFR 1 .136(a). In no event, however, may a reply be timely filed 
after SIX (6) MONTHS from the mailing date of this communication. 
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earned patent term adjustment. See 37 CFR 1.704(b). 

Status 

1)E3 Responsive to communication(s) filed on 28 March 2005 . 
2a)(3 This action is FINAL. 2b)D This action is non-final. 

3) D Since this application is in condition for allowance except for formal matters, prosecution as to the merits is 

closed in accordance with the practice under Ex parte Quayle, 1935 CD. 11, 453 O.G. 213. 

Disposition of Claims 

4) ^ Claim(s) 1.7.9.31.37.39 and 73-106 is/are pending in the application. 

4a) Of the above claim(s) is/are withdrawn from consideration. 

5) D Claim(s) is/are allowed. 

6M Claim(s) 1.7.9.31.37.39.73-81.85-90.98 and 102-106 is/are rejected. 

7) M Claim(s) 82-84 and 99-101 is/are objected to. 

8) D Claim(s) are subject to restriction and/or election requirement. 

Application Papers 

9) D The specification is objected to by the Examiner. 

10)S The drawing(s) filed on 24 April 2000 is/are: a)[3 accepted or b)D objected to by the Examiner. 

Applicant may not request that any objection to the drawing(s) be held in abeyance. See 37 CFR 1 .85(a). 

Replacement drawing sheet(s) including the correction is required if the drawing(s) is objected to. See 37 CFR 1.121(d). 
11 )□ The oath or declaration is objected to by the Examiner. Note the attached Office Action or form PTO-152. 
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application from the International Bureau (PCT Rule 17.2(a)). 
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DETAILED ACTION 

Response to Arguments 
The amendment filed March 28, 2005 has been entered and made of record. 
Applicant's arguments on page 14, line 9 thru page 16, line 8, filed March 28, 2005, with 

regard to claims 1 and 3 1 have been considered but are moot in view of the new ground(s) of 

rejection. 

Applicant's filed March 28, 2005 with regard to the rejections of claims 7, 9, 37 and 39 
have been fully considered but they are not persuasive. 

On page 17, lines 2-5, applicant states, "no motivation to combine Lim and Ratakonda is 
presented because Lim discloses, inter alia, general image indexing (i.e., image can be different 
from on another) as opposed to the video summarization (i.e., images are related to one another 
temporally) recited in the claimed invention". The examiner respectfully disagrees with this 
interpretation. The Lim reference does not explicitly state the images to be indexed cannot be 
temporally related, furthermore, images that are related to one another temporally often do have 
some amount pf change or difference between them. Applicant states, "Lim summarizes each 
image for purposes of a query-based search and therefore does not "summarize" a group of 
images." Lim discloses, "Embodiments of the invention provide a system for creating visual 
"keywords" from visual documents. The visual keywords are prototypical, visual entities present 
in the content of a given distribution of visual documents, which can include digital images and 
video-shot keyframes " (emphasis added; column 6, line 18- 30). A video sequence is composed 
of a plurality of digital images. In addition, Ratakonda discloses, "A video sequence may be 
indexed on the basis of its summary frames using techniques developed for still images. Multiple 
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levels provide flexibility towards indexing at varying detail level " (col. 4, lines 17-20) Examiner 
believes that at least these statements put Lim in the proper context of the instant application and 
the references of Lim and Ratakonda are combinable. 

Claim Rejections - 35 USC §103 
The following is a quotation of 35 U.S. C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office action: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in 
section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are 
such that the subject matter as a whole would have been obvious at the time the invention was made to a person 
having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the 
manner in which the invention was made. 

Claims 1, 7, 9, 31, 37, 39, 73-79, 85-89, 90-96, and 102-106 are rejected under 35 U.S.C. 
103(a) as being unpatentable over the combination of U.S. Patent 5,956,026 to Ratakonda and 
U.S. Patent 6,574,378 to Lim. 

As to claim 1, Ratakonda discloses a method for summarizing a content of an input video 
sequence (col. 3, lines 21-29 and 63), said method comprising: 

(a) computing a feature vector for each frame in a set of frames from said input video 
sequence (col. 4, lines 58-54; Image color histograms, i.e., color distributions, constituting 
representative feature vectors of the video frames are computed.). 

Ratakonda does not disclose (b) applying singular value decomposition to a matrix 
comprised of said feature vectors and projecting the matrix on a refined feature space 
representation, wherein positions of said projections on said refined feature space representation 
represent approximations of visual changes in said set of frames from said input video. 
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Lim discloses a method and apparatus for indexing and retrieving images using visual 
keywords wherein during indexing (or retrieval) of a visual document, a spatial aggregation map 
(SAM) of occurrences of visual tokens is created which represents a visual-content signature for 
the visual document (column 9, line 14-28). Subsequently, the SAM is input to a singular-value- 
decomposition (SVD) based coding module to produce a refined feature (reduced 
dimensionality) space representing a coded description of a visual document wherein a frequency 
matrix X associates visual keywords and visual documents by concatenating linearized SAM 
vectors of visual documents as column vectors of the matrix X (column 9, line 29-37). It would 
have been obvious to one of ordinary skill in the art at the time the invention was made to obtain 
a refined feature space using a singular value decomposition of an input video sequence as taught 
by Lim in order to reduce the dimensionality and possibly the noise in the spatial aggregation 
map to produce a coded description of a visual document (column 9, line 29-33). 

Lim further discloses positions of said projections on said refined feature space 
representation represent approximations of visual changes in said set of frames from said input 
video (col. 9, lines 53-60). Lim teaches similarity matching between two visual documents 
(frames) using a distance measurement. This distance measurement is calculated from the 
positions of the visual documents in the refined feature space. The measurement is disclosed as 
similarity matching but could also obviously be interpreted as a measure of difference between 
the visual documents. 

As to claim 3 1, Ratakonda further discloses the method can implemented using a 
computer program and computer (col. 15, lines 37-38 and 51-53). It is well known that computer 
programs are stored on computer readable mediums such as CD-ROM, floppy disks, as well as 
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magnetic tape, ROM, etc. Lim also discloses a computer readable medium containing a program 
for causing a computer to perform the method of the invention (col. 14, lines 7-63). As to the 
remainder of the claim, arguments analogous to those presented above for claim 1 are applicable 
to claim 31. 

As to claims 7 and 37, Ratakonda further discloses that minimization (to a refined feature 
space) is performed using frames (keyframes) selected with a fixed interval (equispaced) from 
said input video sequence (col. 7, line 42-49). 

As to claim 9 and 39, Lim discloses that said singular value decomposition produces a 
matrix (X), each column of said matrix representing a frame (SAM vectors of visual documents) 
in a refined feature (reduced dimensionality) space corresponding to a frame in said input video 
sequence (column 7, line 33-44). 

As to claims 73 and 90, Ratakonda further discloses the feature vectors are computed 
using a color histogram that outputs a histogram vector (col. 4, lines 48-54). 

As to claims 74 and 91, Ratakonda further discloses the histogram vector is indicative of 
a spatial distribution of colors in said each of said frames (col 4, lines 48-54). 

As to claims 75 and 92, Ratakonda further discloses each of said frames (finest level 
keyframes) is divided into a plurality of blocks, each of said plurality of blocks being represented 
by a histogram in a color space indicative of a distribution of colors within each of said blocks 
(col. 11, lines 1-17; col. 14, lines 15-47). 

As to claims 76 and 93, Ratakonda further discloses that each of said plurality of frames 
(finest level keyframes) is divided into a plurality of blocks and each said histogram vector 
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comprises a plurality of histograms in a color space, each of said plurality of histograms 
corresponding to one of said plurality of blocks (column 11, line 1-17; column 14, line 15-47). 

As to claims 77 and 94, Lim further discloses clustering said frames of said input video 
sequence based upon positions of said projections on said refined feature space representation 
(col. 7, lines 37-55). 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to cluster said frames based upon positions of said projections on said refined feature 
space representation as taught by Lim because it can be implemented unsupervised and ensures 
quality clustering with minimum distances between like/similar image frames. 

As to claims 78 and 95, Ratakonda further discloses selecting a frame from each cluster 
to serve as a keyframe in a summarization of said input video sequence (col. 4, lines 54-63). 

As to claims 79 and 96, Ratakonda further discloses selecting a frame comprises locating 
a frame with a feature vector that projects into a singular value that is most representative of 
other singular values of the cluster (col. 10, lines 6-7, 35-38; Ratakonda teaches assigning the 
centroid (or mean) histogram as the representative vector for each set of vectors and the frame 
whose histogram vector is closest to the representative vector is selected as the first keyframe.). 

As to claims 85 and 102, Ratakonda further discloses clustering of said frames further 
comprises using a position of the most static shot (minimized action) of said input video 
sequence to compute a value as a threshold during the cluster of said frames (col. 6 lines 26 thru 
col. 7, line 33; Figure 6). 
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Claims 80 and 81are rejected under 35 U.S.C. 103(a) as being unpatentable over the 
combination of Ratakonda and Lim as applied to claims 78 and 95 above, and further in view of 
U.S. patent 6,535,639 to Uchihachi et al. (Uchihachi). 

As to claims 80 and 97 Ratakonda does not disclose using said clustered frames to output 
a motion video representative of a summary of said input video sequence. Uchihachi discloses 
automatic video summarization using a measure of shot importance and a frame-packing method 
wherein a video may be modified during playback to emphasize the measure of shot importance 
in order to de-emphasize less important shots, or skip shots entirely that are less important than a 
predetermined threshold, resulting in a summary or "video skim" (column 9, line 53-60). It 
would have been obvious to one of ordinary skill in the art at the time the invention was made to 
output a motion video representative of a summary of an input video sequence as taught by 
Uchihachi in order to emphasize the measure of shot importance by de-emphasizing or skipping 
less important shots and also allowing the user to generate a personal video summary (column 9, 
line 53-60). 

Claims 81 and 98 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Ratakonda, Lim and Uchihachi as applied to claims 80 and 97 above, and further in view of U.S. 
Patent 5,805,733 to Wang et al. (Wang). 

As to claims 81 and 98, Ratakonda, Lim and Uchihachi to not disclose the input video 
sequence summary is composed according to a time-length parameter and a minimum display 
time parameter. However, Wang discloses a normalized lower bound for the time duration of a 
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scene (minimum display time parameter) and a normalized time duration of all scenes (time- 
length parameter) (col. 5, lines 17-40) in a method for summarizing scenes in a video sequence. 

It would have been obvious to one of ordinary skill in the art at the time the invention 
was made to compose the input video sequence summary according to a total length parameter 
and a minimum display time parameter in order to determine whether a scene is of relative 
importance to the input video and whether it is selected for use in the video summary. 

Claims 86-89 and 103-106 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over the combination of Ratakonda and Lim as applied to claims 85 and 102 above, and further 
in view of U.S. Patent 6, 122,628 to Castelli et al. (Castelli). 

As to claims 86 and 103, Ratakonda further discloses clustering of said frames further 
comprises computing a content value (keyframe similarity base on histogram feature vector) 
(col. 9, lines 13-56) to cluster the remaining frames (keyframes representing video shots) (col. 9, 
line 44 thru col. 10 line 42) by: 

sorting said feature vectors in said refined feature space according to a distance of each of 
said vectors to an origin (representative vector) of said refined feature space representation 
(column 9, line 49-column 10, line 20); 

selecting a vector among said sorted feature vectors which is closest to an origin 
(representative vector) of said refined feature space representation and including said selected 
vector into a first cluster (column 10, line 35-37); 

clustering said plurality of sorted feature vectors in said refined feature space 
representation into a plurality of clusters according to a distance between each of said plurality of 
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sorted feature vectors and feature vectors in each of said plurality of clusters and an amount of 
information in each of said plurality of clusters (column 10, line 37-42). Ratakonda does not 
explicitly disclose that the vectors are sorted in ascending order. 

Castelli discloses multidimensional data clustering and dimension reduction for indexing 
and searching wherein eigenvalues (characteristic values of a transformation matrix produced by 
singular value decomposition; column 11, line 9-16) are sorted by decreasing magnitude and a 
subset of ordered eigenvalues containing the largest eigenvalues are selected according to a 
selection criterion (column 11, line 25-3 1). Castelli sorts in decreasing order, but it would have 
been obvious to sort in ascending order, depending on the needs of the system. It would have 
been obvious to one of ordinary skill in the art at the time the invention was made to sort a 
plurality of vectors in said refined feature space in ascending order as taught by Castelli in order 
to aid in the selection of the vectors (eigenvalues) according to a user-specified selection 
criterion (column 1 1, line 28-40). 

As to claims 87 and 104, Ratakonda further discloses that said plurality of sorted feature 
vectors are clustered into said plurality of clusters such that said amount of information (action) 
in each of said plurality of clusters does not exceed an amount of information (action) in said 
first cluster (column 6, line 45-60). 

As to claims 88 and 105, Ratakonda discloses that said first cluster is composed of frames 
(keyframes) based on a distance variation between said frames (column 8, line 52-63) and an 
average distance between frames in said first cluster (same as the distance since the keyframes 
are equispaced; column 7, line 45-49). 
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As to claims 89 and 106, Ratakonda discloses that each of said plurality of clusters is 
composed of frames (keyframes) based on a distance variation between said frames (column 8, 
line 52-63) and an average distance between frames in said each of said plurality of clusters 
(same as the distance since the keyframes are equispaced; column 7, line 45-49). 

Allowable Subject Matter 

Claims 82-84 and 99-101 are objected to as being dependent upon a rejected base claim, 
but would be allowable if rewritten in independent form including all of the limitations of the 
base claim and any intervening claims. 

The following is a statement of reasons for the indication of allowable subject matter: 
The features as explicitly recited in claims 82 and 99 and in combination with the base claims 
and any intervening claims, are neither disclosed nor suggested by the prior art of record. Claims 
83 and 84 depend from claim 82. Claims 100 and 101 depend from claim 99 

Conclusion 

Applicant's amendment necessitated the new ground(s) of rejection presented in this 
Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). 
Applicant is reminded of the extension of time policy as set forth in 37 CFR 1 .136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within TWO 
MONTHS of the mailing date of this final action and the advisory action is not mailed until after 
the end of the THREE-MONTH shortened statutory period, then the shortened statutory period 
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will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 
CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, 
however, will the statutory period for reply expire later than SIX MONTHS from the date of this 
final action. 



Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Anthony Mackowey whose telephone number is (571) 272-7425. 
The examiner can normally be reached on M-F 9:00-6:00. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Amelia Au can be reached on (571) 272-7414. The fax phone number for the 
organization where this application or proceeding is assigned is 703-872-9306. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published applications 
may be obtained from either Private PAIR or Public PAIR. Status information for unpublished 
applications is available through Private PAIR only. For more information about the PAIR 
system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR 
system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). 
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